Community consensus on core open science practices to monitor in biomedicine.

The state of open science needs to be monitored to track changes over time and identify areas to create interventions to drive improvements. In order to monitor open science practices, they first need to be well defined and operationalized. To reach consensus on what open science practices to monitor at biomedical research institutions, we conducted a modified 3-round Delphi study. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices, and for round 3, we hosted two half-day virtual meetings to discuss and vote on items that had not reached consensus. Ultimately, participants reached consensus on 19 open science practices. This core set of open science practices will form the foundation for institutional dashboards and may also be of value for the development of policy, education, and interventions.

Rapid antigen-based and rapid molecular tests for the detection of SARS-CoV-2: a rapid review with network meta-analysis of diagnostic test accuracy studies.

BACKGROUND: The global spread of COVID-19 created an explosion in rapid tests with results in < 1 hour, but their relative performance characteristics are not fully understood yet. Our aim was to determine the most sensitive and specific rapid test for the diagnosis of SARS-CoV-2. METHODS: Design: Rapid review and diagnostic test accuracy network meta-analysis (DTA-NMA). ELIGIBILITY CRITERIA: Randomized controlled trials (RCTs) and observational studies assessing rapid antigen and/or rapid molecular test(s) to detect SARS-CoV-2 in participants of any age, suspected or not with SARS-CoV-2 infection. INFORMATION SOURCES: Embase, MEDLINE, and Cochrane Central Register of Controlled Trials, up to September 12, 2021. OUTCOME MEASURES: Sensitivity and specificity of rapid antigen and molecular tests suitable for detecting SARS-CoV-2. Data extraction and risk of bias assessment: Screening of literature search results was conducted by one reviewer; data abstraction was completed by one reviewer and independently verified by a second reviewer. Risk of bias was not assessed in the included studies. DATA SYNTHESIS: Random-effects meta-analysis and DTA-NMA. RESULTS: We included 93 studies (reported in 88 articles) relating to 36 rapid antigen tests in 104,961 participants and 23 rapid molecular tests in 10,449 participants. Overall, rapid antigen tests had a sensitivity of 0.75 (95% confidence interval 0.70-0.79) and specificity of 0.99 (0.98-0.99). Rapid antigen test sensitivity was higher when nasal or combined samples (e.g., combinations of nose, throat, mouth, or saliva samples) were used, but lower when nasopharyngeal samples were used, and in those classified as asymptomatic at the time of testing. Rapid molecular tests may result in fewer false negatives than rapid antigen tests (sensitivity: 0.93, 0.88-0.96; specificity: 0.98, 0.97-0.99). The tests with the highest sensitivity and specificity estimates were the Xpert Xpress rapid molecular test by Cepheid (sensitivity: 0.99, 0.83-1.00; specificity: 0.97, 0.69-1.00) among the 23 commercial rapid molecular tests and the COVID-VIRO test by AAZ-LMB (sensitivity: 0.93, 0.48-0.99; specificity: 0.98, 0.44-1.00) among the 36 rapid antigen tests we examined. CONCLUSIONS: Rapid molecular tests were associated with both high sensitivity and specificity, while rapid antigen tests were mainly associated with high specificity, according to the minimum performance requirements by WHO and Health Canada. Our rapid review was limited to English, peer-reviewed published results of commercial tests, and study risk of bias was not assessed. A full systematic review is required. REVIEW REGISTRATION: PROSPERO CRD42021289712.

Ensuring an Accurate Scientific Record in an Era of Pre-print Servers.

BACKGROUND: Pre-prints have become an increasing part of the biomedical landscape. For example, during the first month of operation, July 2019, medRxiv received 176 submissions, one year later, in June 2020, including the first few months of COVID-19, it received 1866 submissions. The current relevant question is how to ensure an accurate scientific record given that there may be important differences between a pre-print and the peer-reviewed publication. METHODS: Based upon the experience of the authors, conversations with editors, and a focused selective review of the literature, including the recommendations of some professional groups, a limited number of practical recommendations were formulated. RESULTS: Peer-reviewed journals should request that authors indicate if the submitted manuscript has been posted on a pre-print server; ensure this is noted in the article if it is published by including the digital object identifier (DOI); and detail any major differences in the conclusions between the pre-print and the article. Pre-print servers should ensure that all content is marked as not peer-reviewed and be prepared to retract any pre-print that is fundamentally flawed within days that could influence clinical or public health recommendations that have therapeutic implications. CONCLUSION: Authors, those responsible for pre-print servers, and editors of peer- reviewed journals are responsible for ensuring an accurate scientific record.

Lessons from the COVID-19 pandemic and recent developments on the communication of clinical trials, publishing practices, and research integrity: in conversation with Dr. David Moher.

BACKGROUND: The torrent of research during the coronavirus (COVID-19) pandemic has exposed the persistent challenges with reporting trials, open science practices, and scholarship in academia. These real-world examples provide unique learning opportunities for research methodologists and clinical epidemiologists-in-training. Dr. David Moher, a recognized expert on the science of research reporting and one of the founders of the Consolidated Standards of Reporting Trials (CONSORT) statement, was a guest speaker for the 2021 Hooker Distinguished Visiting Professor Lecture series at McMaster University and shared his insights about these issues. MAIN TEXT: This paper covers a discussion on the influence of reporting guidelines on trials and issues with the use of CONSORT as a measure of quality. Dr. Moher also addresses how the overwhelming body of COVID-19 research reflects the "publish or perish" paradigm in academia and why improvement in the reporting of trials requires policy initiatives from research institutions and funding agencies. We also discuss the rise of publication bias and other questionable reporting practices. To combat this, Dr. Moher believes open science and training initiatives led by institutions can foster research integrity, including the trustworthiness of researchers, institutions, and journals, as well as counter threats posed by predatory journals. He highlights how metrics like journal impact factor and quantity of publications also harm research integrity. Dr. Moher also discussed the importance of meta-science, the study of how research is carried out, which can help to evaluate audit and feedback systems and their effect on open science practices. CONCLUSION: Dr. Moher advocates for policy to further improve the reporting of trials and health research. The COVID-19 pandemic has exposed how a lack of open science practices and flawed systems incentivizing researchers to publish can harm research integrity. There is a need for a culture shift in assessing careers and "productivity" in academia, and this requires collaborative top-down and bottom-up approaches.

Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement.

OBJECTIVE: To develop a reporting guideline for overviews of reviews of healthcare interventions. DESIGN: Development of the preferred reporting items for overviews of reviews (PRIOR) statement. PARTICIPANTS: Core team (seven individuals) led day-to-day operations, and an expert advisory group (three individuals) provided methodological advice. A panel of 100 experts (authors, editors, readers including members of the public or patients) was invited to participate in a modified Delphi exercise. 11 expert panellists (chosen on the basis of expertise, and representing relevant stakeholder groups) were invited to take part in a virtual face-to-face meeting to reach agreement (≥70%) on final checklist items. 21 authors of recently published overviews were invited to pilot test the checklist. SETTING: International consensus. INTERVENTION: Four stage process established by the EQUATOR Network for developing reporting guidelines in health research: project launch (establish a core team and expert advisory group, register intent), evidence reviews (systematic review of published overviews to describe reporting quality, scoping review of methodological guidance and author reported challenges related to undertaking overviews of reviews), modified Delphi exercise (two online Delphi surveys to reach agreement (≥70%) on relevant reporting items followed by a virtual face-to-face meeting), and development of the reporting guideline. RESULTS: From the evidence reviews, we drafted an initial list of 47 potentially relevant reporting items. An international group of 52 experts participated in the first Delphi survey (52% participation rate); agreement was reached for inclusion of 43 (91%) items. 44 experts (85% retention rate) completed the second Delphi survey, which included the four items lacking agreement from the first survey and five new items based on respondent comments. During the second round, agreement was not reached for the inclusion or exclusion of the nine remaining items. 19 individuals (6 core team and 3 expert advisory group members, and 10 expert panellists) attended the virtual face-to-face meeting. Among the nine items discussed, high agreement was reached for the inclusion of three and exclusion of six. Six authors participated in pilot testing, resulting in minor wording changes. The final checklist includes 27 main items (with 19 sub-items) across all stages of an overview of reviews. CONCLUSIONS: PRIOR fills an important gap in reporting guidance for overviews of reviews of healthcare interventions. The checklist, along with rationale and example for each item, provides guidance for authors that will facilitate complete and transparent reporting. This will allow readers to assess the methods used in overviews of reviews of healthcare interventions and understand the trustworthiness and applicability of their findings.

Reporting and data sharing level for COVID-19 vaccine trials: A cross-sectional study.

BACKGROUND: The results and data availability of vaccine trials directly affect the decisions of healthcare providers, the public, and policymakers as to whether the vaccine should be applied. However, the reporting and data sharing level of COVID-19 vaccine studies are not clear. METHODS: A cross-sectional study was conducted. A systematic search up to 9 May 2021 in 12 databases and an updated search to 6 July 2021 were conducted in the Cochrane Living Systematic Review and Network Meta-Analysis database to identify COVID-19 vaccine trials. The basic characteristics of included trials were summarized. The reporting level was assessed according to the CONSORT checklist. The data sharing level was assessed by open science practices. Types of incomplete reporting including protocol deviation, lack of primary outcomes clarity, and the omission of harms were analyzed. FINDINGS: Finally, thirty-six COVID-19 vaccine articles reporting on 40 randomized controlled trials were included in this analysis. Based on the CONSORT checklist, the mean reporting score was 29.7 [95% confidence interval 28.7, 30.7]. Thirty-one articles (31/36, 86.1%) had data sharing statements, twenty-five articles (25/36, 69.4%) provided access to the source data. Twenty-seven articles (27/36, 75.0%) had protocol deviation, lack of primary outcomes clarity, or the omission of harms. INTERPRETATION: The reporting and data sharing level of COVID-19 vaccine trials were not optimal. We hope that the reporting and data sharing of future trials will be improved. We recommend establishing a comprehensive, accurate data sharing system for future vaccine trials. FUNDING: This work was supported by the National Key R&D Program of China (2019YFC1710400; 2019YFC1710403).

Medical journal requirements for clinical trial data sharing: Ripe for improvement.

Florian Naudet and co-authors discuss strengthening requirements for sharing clinical trial data.

Guidelines for Reporting Trial Protocols and Completed Trials Modified Due to the COVID-19 Pandemic and Other Extenuating Circumstances: The CONSERVE 2021 Statement.

Importance: Extenuating circumstances can trigger unplanned changes to randomized trials and introduce methodological, ethical, feasibility, and analytical challenges that can potentially compromise the validity of findings. Numerous randomized trials have required changes in response to the COVID-19 pandemic, but guidance for reporting such modifications is incomplete. Objective: As a joint extension for the CONSORT and SPIRIT reporting guidelines, CONSERVE (CONSORT and SPIRIT Extension for RCTs Revised in Extenuating Circumstances) aims to improve reporting of trial protocols and completed trials that undergo important modifications in response to extenuating circumstances. Evidence: A panel of 37 international trial investigators, patient representatives, methodologists and statisticians, ethicists, funders, regulators, and journal editors convened to develop the guideline. The panel developed CONSERVE following an accelerated, iterative process between June 2020 and February 2021 involving (1) a rapid literature review of multiple databases (OVID Medline, OVID EMBASE, and EBSCO CINAHL) and gray literature sources from 2003 to March 2021; (2) consensus-based panelist meetings using a modified Delphi process and surveys; and (3) a global survey of trial stakeholders. Findings: The rapid review yielded 41 673 citations, of which 38 titles were relevant, including emerging guidance from regulatory and funding agencies for managing the effects of the COVID-19 pandemic on trials. However, no generalizable guidance for all circumstances in which trials and trial protocols might face unanticipated modifications were identified. The CONSERVE panel used these findings to develop a consensus reporting guidelines following 4 rounds of meetings and surveys. Responses were received from 198 professionals from 34 countries, of whom 90% (n = 178) indicated that they understood the concept definitions and 85.4% (n = 169) indicated that they understood and could use the implementation tool. Feedback from survey respondents was used to finalize the guideline and confirm that the guideline's core concepts were applicable and had utility for the trial community. CONSERVE incorporates an implementation tool and checklists tailored to trial reports and trial protocols for which extenuating circumstances have resulted in important modifications to the intended study procedures. The checklists include 4 sections capturing extenuating circumstances, important modifications, responsible parties, and interim data analyses. Conclusions and Relevance: CONSERVE offers an extension to CONSORT and SPIRIT that could improve the transparency, quality, and completeness of reporting important modifications to trials in extenuating circumstances such as COVID-19.

Benefits and obstacles to cell therapy in neonates: The INCuBAToR (Innovative Neonatal Cellular Therapy for Bronchopulmonary Dysplasia: Accelerating Translation of Research).

Cell-based therapies hold promise to substantially curb complications from extreme preterm birth, the main cause of death in children below the age of 5 years. Exciting preclinical studies in experimental neonatal lung injury have provided the impetus for the initiation of early phase clinical trials in extreme preterm infants at risk of developing bronchopulmonary dysplasia. Clinical translation of promising therapies, however, is slow and often fails. In the adult population, results of clinical trials so far have not matched the enticing preclinical data. The neonatal field has experienced many hard-earned lessons with the implementation of oxygen therapy or postnatal steroids. Here we briefly summarize the preclinical data that have permitted the initiation of early phase clinical trials of cell-based therapies in extreme preterm infants and describe the INCuBAToR concept (Innovative Neonatal Cellular Therapy for Bronchopulmonary Dysplasia: Accelerating Translation of Research), an evidence-based approach to mitigate the risk of translating advanced therapies into this vulnerable patient population. The INCuBAToR addresses several of the shortcomings at the preclinical and the clinical stage that usually contribute to the failure of clinical translation through (a) systematic reviews of preclinical and clinical studies, (b) integrated knowledge transfer through engaging important stakeholders early on, (c) early economic evaluation to determine if a novel therapy is viable, and (d) retrospective and prospective studies to define and test ideal eligibility criteria to optimize clinical trial design. The INCuBAToR concept can be applied to any novel therapy in order to enhance the likelihood of success of clinical translation in a timely, transparent, rigorous, and evidence-based fashion.

Retraction of COVID-19 Pharmacoepidemiology Research Could Have Been Avoided by Effective Use of Reporting Guidelines.

INTRODUCTION: Two recent high-profile publications (and subsequent retractions) of pharmacoepidemiology studies reporting the effectiveness and risk of hydroxychloroquine in COVID-19 patients received international media attention. Transparent and complete reporting of these studies could have provided peer reviewers and editors with sufficient information to question the methods used and the validity of results. Since these studies used routinely collected health data, the guidelines for the REporting of studies Conducted using Observational Routinely collected health Data (RECORD) should have been applied to ensure complete reporting of the research. METHODS: We evaluated the two retracted articles for completeness of reporting using the RECORD for Pharmacoepidemiology (RECORD-PE) checklist, which includes the checklists for the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) and RECORD. We compared the proportion of STROBE, RECORD and RECORD-PE items adequately reported using Chi-squared statistics. RESULTS: In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%) and 9.5 of 15 RECORD-PE items (63.3%)(χ2 = 14.839, P <0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%) (χ2 = 11.668, P = 0.003). Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, validation of the coding or algorithms, a description of the underlying database population and the accuracy of data linkage methods. DISCUSSION: While STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Reporting guidelines should be effectively implemented in order for transparency and completeness of research manuscripts, allowing for adequate evaluation by editors and peer reviewers.

COVID-19 and the research scholarship ecosystem: help!

OBJECTIVES: Data sharing practices remain elusive in biomedicine. The COVID-19 pandemic has highlighted the problems associated with the lack of data sharing. The objective of this article is to draw attention to the problem and possible ways to address it. STUDY DESIGN AND SETTING: This article examines some of the current open access and data sharing practices at biomedical journals and funders. In the context of COVID-19 the consequences of these practices is also examined. RESULTS: Despite the best of intentions on the part of funders and journals, COVID-19 biomedical research is not open. Academic institutions need to incentivize and reward data sharing practices as part of researcher assessment. Journals and funders need to implement strong polices to ensure that data sharing becomes a reality. Patients support sharing of their data. CONCLUSION: Biomedical journals, funders and academic institutions should act to require stronger adherence to data sharing policies.

Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.

Importance: Convalescent plasma is a proposed treatment for COVID-19. Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs). Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021. Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting. Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation. Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events. Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10 722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences. Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.

Publication by association: how the COVID-19 pandemic has shown relationships between authors and editorial board members in the field of infectious diseases.

During the COVID-19 pandemic, the rush to scientific and political judgements on the merits of hydroxychloroquine was fuelled by dubious papers which may have been published because the authors were not independent from the practices of the journals in which they appeared. This example leads us to consider a new type of illegitimate publishing entity, 'self-promotion journals' which could be deployed to serve the instrumentalisation of productivity-based metrics, with a ripple effect on decisions about promotion, tenure and grant funding, but also on the quality of manuscripts that are disseminated to the medical community and form the foundation of evidence-based medicine.